chore: add agent-run dashboard flow checks and Magellan explorer - #8321

DES-432

Summary

Two agent-driven tools for catching dashboard UX drift. Neither blocks merges. Both run their agents on GitHub Agentic Workflows (gh-aw) in its AWF sandbox, not on custom sandboxing.

Flow checks (e2e/flows/, .github/workflows/flow-checks.md) protect known flows:

• Each flow is a plain-English story.md (Given / When / Then, with every Then proven by an /rpc/ call, not by text on screen) next to a Playwright flow.spec.ts that a Claude Code agent compiled from it.
• On PRs that touch the dashboard, ordinary workflow steps boot the stack and run mise run test:flows. If every flow passes, the run ends there with no LLM.
• If a flow fails, a Claude Code agent re-walks each failing story (.github/prompts/heal-flow.md) inside the AWF sandbox and returns one verdict per flow:
  • healed: the UI moved; it rewrites the script, which is attached to the run as the healed-flows artifact.
  • broken: the flow can't be finished.
  • changed: the PR changes the flow on purpose, so the story needs an edit.
• It posts one PR comment (a gh-aw safe output) with the verdicts and the commands to replay and heal each flow locally (mise run test:flows flows/<flow>/, mise run flows:heal <flow>).
• Three starter flows: server-from-source (fully repeatable), assistant-create (needs an LLM key; skipped without one), catalog-connect (uses the live registry).

Magellan (e2e/magellan/) is a nightly explorer for problems no story covers. It runs against production, in a dedicated integration org, as a bot user. Its gh-aw workflow lives in speakeasy-api/gram-infra (.github/workflows/magellan.md), because gh-aw uploads every run's agent transcript as an artifact and this repo is public. That workflow checks out this repo's main for the prompt (.github/prompts/magellan-explore.md) and the filer.

• A finding needs hard evidence (console error, failed RPC, dead end, race, or a control with no accessible name) and must reproduce twice.
• Findings leave the agent only through a file_finding safe output. A separate job, holding the Linear key, runs mise run magellan:file --agent-output, which validates each finding, deduplicates by fingerprint so a bug seen on several nights stays one Linear issue, and puts agent text into Linear only inside code blocks.
• mise run magellan:explore still works against the local stack, unsandboxed, for trying out prompt changes.

How the agents are contained

All of this comes from gh-aw and AWF, not from code in this PR:

• The agent runs in AWF's container. Its egress is limited by AWF's firewall: for flow checks, the stack's dashboard, server and dev-idp ports on the runner (docker-sudo-iptables profile with allow-host-ports); for Magellan, app.getgram.ai.
• ANTHROPIC_API_KEY is excluded from the agent's environment; AWF's API proxy holds it.
• The agent writes back only through safe outputs (PR comment, artifact, Linear filing job), each run in a separate job with its own token or secret.
• gh-aw runs threat detection on the agent's output before any safe output is applied, and redacts the workflow's secrets from uploaded logs.
• Compiled with gh-aw v1.2.1 in action mode, so every gh-aw action is pinned to github/gh-aw-actions@406c7719e31d7fe354224d4bb8fe3d4aeb47da1c and every container image to a digest.

Security review (gh-aw compile)

gh aw compile asked for a review of what this compilation adds:

• ANTHROPIC_API_KEY (secret): the engine credential. Excluded from the agent container (--exclude-env) and held by AWF's API proxy. Safe.
• ATLAS_TOKEN (secret): needed by ./zero --agent to run migrations. Set only on the "Boot the stack" step, which runs outside the sandbox before the agent starts. Same use as wake-stack-e2e.yml. Safe.
• jdx/mise-action (action): pinned to 9149ea85001c7435d5a66bb127d6a1b6227cb0a5 (v5.0.0), the same pin the repo already uses elsewhere. Safe.

.github/zizmor.yml ignores three low audits on the generated flow-checks.lock.yml (it can't carry inline comments); the reasons are in the file. .oxfmtrc.json ignores *.lock.yml, since reformatting a lock file would break gh-aw's source check.

Motivation

Changes can quietly break whether a user can finish a flow, and nothing catches it today. Running an agent as the per-PR gate would be slow, costly and too willing to say "pass". Here the agent writes and repairs scripts, the scripts do the gating, and the open-ended exploring runs nightly instead.

Findings from the spike:

• Heal runs take 6–16 minutes each, so the agent runs only when a replay fails.
• Flows that depend on an LLM or a live upstream server are better suited to the nightly run than to every PR.
• Magellan's first local run reported findings that came from the local stack (feature gates off, state from earlier runs). That is why Magellan targets production.

Setup before first run

• This repo: no new secrets (ANTHROPIC_API_KEY and ATLAS_TOKEN exist).
• speakeasy-api/gram-infra: a magellan environment restricted to main, with MAGELLAN_ORG, MAGELLAN_SESSION, ANTHROPIC_API_KEY and MAGELLAN_LINEAR_API_KEY (details in that workflow's header). Its filing job runs mise run magellan:file --agent-output, which exists only once this PR merges.

Gutternote review steps

No user-visible change: this PR adds tooling only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PXf4pFQMBT5vL8roKYJdbW